Add NumPy-like add and multiply for CLR arrays - #2110

I guess some comment is better than silence for a few days while I look through this, so here are my first ones:

Thanks! I really appreciate it. It felt like silence since the original proposal, I didn't know whether you were even positive about the whole idea of a NumPy-like interface for System.Array.

I tried this one before checking the examples 😄.

Array[bool]([True]).add(Array[bool]([False]), out=Array[bool]([False])) # [False]

The instance is ignored and out out gets bound to the 2nd argument. Seems like a core argument binding issue.

Touché. This is clearly wrong, but Array was not supposed to have instance add at all. This works:

Array.add(Array[bool]([True]), (Array[bool]([False])), out=Array[bool]([False])) # [True]

Also in NumPy, there is no instance add on ndarray; there is only a static add on the module:

np.add(np.array([True], dtype=bool), np.array([False], dtype=bool), out=np.array([False], dtype=bool)) # [True]
np.array([True], dtype=bool).add(np.array([False], dtype=bool), out=np.array([False], dtype=bool))  # AttributeError

I'm thinking if creating a new module for all those NumPy-like functions, keeping only operators on System.Array, and maybe dtype. I don't want to crowd the list of methods attached to the type; the new functions should be only visible if somebody wants them explicitly. Also it could go to a separate DLL, or even an IronPython extension. Any ideas for names? System.Array.NDArray or _ndarray come to my mind.

It looks like you're deferring to IronPython's binary operations to get the results

Yes, that was the idea.

Anyway, it's just not as clear cut as numpy which doesn't grow out of its type.

It is not, indeed.

Here's my take on this: Let's limit the discussion to numeric scalars only, it is already complicated enough. In NumPy, there are two disjoint groups of types: Python native 'weak' types: float, int, complex; and NumPy's strong types. There is a whole set of rules how they get promoted (NEP 50), but in short, a weak scalar adapts to the strong operand in kind, size/precision is ignored.

In IronPython, there are also two groups: Python types and the System types, the latter treated as NumPy's strong types. But the sets partially overlap; in that case the type is treated as a Python type, hence 'weak'. With this in mind, the promotion rules cannot be made identical to NumPy, but they should be close (and usable) enough.

I see some problems in the implementation, though:

Array.add(Array[bool]([True]), Array[bool]([True]), out=Array[Int64]([0]))
# should be Array[Int64]([1]), not Array[Int64]([2])
Array.add(Array[Int16]([32767]), Array[Int16]([1]), out=Array[Int64]([0]))
# I'd expect OverflowError, not Array[Int64]((32768))
# by NumPy rules it would be Array[Int64]((-32768)), since it does not overflow and wraps instead.

I will look into the promotion rules in more detail and hopefully a fix.

Maybe? .NET 8/9 will reach end of support soon out so maybe a good time... But on the other hand there are so many things I want to look at I'm not sure where to draw the line. 🐱

Don't draw any line! Just take a snapshot of what the status is and do a release. Every new commit brings some sort of improvement, so a release at any point is better than the previous release. I was hoping for the last release that includes .NET 8, after which we can ditch target net8.0 (and net9.0 in the DLR) for good. (Was also hoping for the new target net11.0, because of a personal interest in runtime async, but it doesn't have to go on NuGet as an official package). My only hope is to get this particular PR (with warnings on old operator use) in before the release.

Ideally, we could have a yearly release, perhaps synchronized with .NET releases. It really doesn't matter that much what commit is being released, as long as it is being released at all. I know that making a release is a chore and if I could help, take me on my offer. Late fall (like now) I should have time.

In IronPython, there are also two groups: Python types and the System types, the latter treated as NumPy's strong types.

Yes, except we're growing things to a bigger type which is not very useful imo. For example:

>>> type(System.Byte(0xff) + System.Byte(1))
<class 'Int16'>

I'm not sure when going to an Int16 would ever be useful. Some more reasonable options would by:

• unchecked overflow (probably my preference)
• OverflowException
• grow to a Python int

and similarly for all the System types (not sure how we'd handle Int32... maybe we could fully switch int to BigInteger but that's a bigger change that would need benchmarking)